Deep Learning and Model Evaluations on Fashion-MNIST Dataset¶

In the second notebook Part 2 - ML and Model Comparisons we've trained three machine learning classifiers using the Fashion-MNIST dataset from the Data Asset Exchange. In this notebook we will use the Keras deep learning Python library to build and train a model that can be used to identify and classify fashion and clothing items.

Table of Contents:¶

• 0. Prerequisites
• 1. Read Data
• 2. Prepare the Data For Training
• 3. Build the Model
• 4. Train the Model
• 5. Test and Evaluate Model Performance
• Authors

0. Prerequisites¶

Before you run this notebook complete the following steps:

• Insert a project token
• Install and import required packages

Insert a project token¶

When you import this project from the Watson Studio Gallery, a token should be automatically generated and inserted at the top of this notebook as a code cell such as the one below:

# @hidden_cell
# The project token is an authorization token that is used to access project resources like data sources, connections, and used by platform APIs.
from project_lib import Project
project = Project(project_id='YOUR_PROJECT_ID', project_access_token='YOUR_PROJECT_TOKEN')
pc = project.project_context

If you do not see the cell above, follow these steps to enable the notebook to access the dataset from the project's resources:

• Click on More -> Insert project token in the top-right menu section

• This should insert a cell at the top of this notebook similar to the example given above.

  If an error is displayed indicating that no project token is defined, follow these instructions.

• Run the newly inserted cell before proceeding with the notebook execution below

Import required packages¶

Install and import the required packages, including pandas, matplotlib, tensorflow, numpy, sklearn and seaborn.

1. Read Data¶

We start by reading in training data from fashion-mnist_train.csv and test data from fashion-mnist_test.csv into arrays. Each row contains a class label and pixel data.

2. Prepare the Data For Training¶

Process item labels¶

Each row in the training and test dataset files contains a label that identifies the clothing item that is depicted by the pixel data. The label is expressed as a number between 0 and 9, and mapped as follows:

• 0 - T-shirt/top
• 1 - Trouser
• 2 - Pullover
• 3 - Dress
• 4 - Coat
• 5 - Sandal
• 6 - Shirt
• 7 - Sneaker
• 8 - Bag
• 9 - Ankle boot

We start by defining a simple mapping between the label number and a textual description, which we'll use to make the output more reader-friendly.

Extract the labels for each image in the dataset, which is the first value in each row.

We use scikit-learn's sklearn.preprocessing.LabelBinarizer to binarize the labels in a one-vs-all fashion.

Each label is now transformed into an array of 0s and 1s, as shown in the example below for the first row in the data set:

Let's make sure that there are no unexpected labels in the training dataset file. The output should only list the numbers 0 through 9.

Process pixel data¶

As we've discussed in the first notebook when we explored the dataset, the grayscale clothing images are 28px by 28px images in size (for a total of 784) and encoded as pixel values. Extract the pixel values and store them in a Numpy array.

The pixel data in each row contain brightness information for each pixel in the 28x28 grayscale image. The pixel brightness is stored as an 8-bit integer, ranging from 0 (typically representing black) to 255 (white). Values in between represent different shades of gray. To normalize the values, we therefore need to divide each pixel data value by 255.

3. Build the Model¶

Keras is a high-level neural networks API that can run on top of TensorFlow, the Microsoft Cognitive Toolkit (CNTK), or Theano. In this notebook we will utilize the Tensorflow tf.keras module.

In Keras, you assemble layers to build models. A model is usually a graph of layers. The most common type of model is a stack of layers: the tf.keras.Sequential model.

Set up the layers¶

The basic building block of a neural network constitutes multiple layers. Layers extract representations from the data fed into them. Generally, most deep learning models consist of multiple simple layers which are chained together. We use three basic layers as building blocks for our classifier - convolution layer, max pooling layer, and dense layer. These building blocks are common starting points for solving vision problems using deep learning as evidenced by their usage in almost all modern day peer to peer publications in this field.

It is beyond the scope of this notebook to explain the chosen layers and how to tune the hyper parameters to improve performance. Simplified speaking, the model extracts features from the image using convolutional and pooling layers and classifies the object using the dense softmax layer.

Print the model summary.

4. Train the Model¶

Now that we've created the model we need to configure it for training.

Configure model training¶

Let's first define the terms loss function and accuracy metric.

A loss function is used to optimize a machine learning algorithm. The loss is calculated on training and validation and its interpretation is based on how well the model is doing in these two sets. It is the sum of errors made for each example in training or validation sets. Loss value implies how poorly or well a model behaves after each iteration of optimization.

In this notebook we choose categorial_crossentropy as loss function, which is best suited for multi-class classification.

An accuracy metric is used to measure the algorithm’s performance in an interpretable way. Accuracy of a model is usually determined after the model parameters and is calculated in the form of a percentage. It is the measure of how accurate your model's prediction is compared to the true data.

While there are many optimizers for the gradient descent and most of them would do the job, we choose the Adam algorithm as it appears to be the most common across common deep learning implementations for such tasks. During training and testing the accuracy metrics will be evaluated.

For information about additional configuration options and details refer to the model.compile() documentation.

Train the model¶

We train the model by calling model.fit(). As input we pass the prepared training data, reserving 33% of the training data for validation, limit training to 5 epochs (an iteration over the entire training data) and use a batch size of 100.

Using this configuration model training should take a couple of minutes. While training is in progress the training time, loss and accuracy metrics are displayed.

You can see how over time loss decreases and accuracy increases. Running training for 5 epochs, the final accuracy is close to 90%.

If you'd like you can change the number of epochs and batch size to observe how accuracy changes. For example increasing the number of epochs to 10 should give you more accurate results!

Keep in mind that training time will increase and more resources will be required to complete the job.

The model.fit() method returns an object that contains a record of loss values and metrics values (which based on our training configuration is accuracy) for training and validation for each epoch.

To make it easier to review the results we plot loss and accuracy for each epoch using the matplotlib visualization library. The history for the validation dataset is labeled as validation and the history of training dataset is labeled as train.

Note that your graphs might look slightly different.

Model accuracy should increase with each epoch and loss decrease. Do note though that you might observe that overfitting has occured if accuracy drops and the loss increases for the validation data, whereas it improves for the training data.

Save the model¶

You can save the trained model for later re-use in a file using model.save(). The saved file includes

• the model architecture, which allows for re-instantiation of the model
• the model weights
• the optimizer state, which allows for resuming of the training where you left off

5. Test and Evaluate Model Performance¶

Now that we've trained the model it is time to test how accurate its predictions are for the unseen data in fashion-mnist_test.csv. We'll:

• Prepare the test data
• Run predictions on the test data
• Sample the prediction results
• Calculate and visualize model performance metrics

Prepare the Test Data¶

Prepare the test data like we've done earlier with the training data.

Run Predictions on Test Data¶

Use model.predict() to predict which clothing items are depicted in the test data set. The method returns an Numpy array of predictions.

Review the Prediction Results¶

Each row in this array contains a numeric score (ranging from 0 and 1) for each of the 10 classes that we trained the model on, indicating the likelihood that the depicted clothing item belongs to the class. For the first image in the test data set the predictions look as follows:

The higher the score, the more confident the model is that the depicted clothing item belongs to the class. Calculate the maximum for each rows to determine what the predicted label is:

Let's take a peek at the predicted label and the correct label for the first clothing item in the test data set:

Ideally the prediction should be correct, but your results might vary. Let's tally up the numbers for the entire test data.

While this breakdown is useful, it doesn't provide any details as to how well the model can identify individual clothing items. In machine learning, a confusion matrix is commonly used to visualize the performance of a classification algorithm.

Calculate the confusion matrix using the correct labels and predicted labels as input, using scikit-learn's sklearn.metrics.confusion_matrix() method. We then use the seaborn visualization library to display thye matrix as a heatmap.

For the Fashion-MNIST data set the output is a 10 by 10 matrix, comprising of one row and column per class. The correct predictions are located in the diagonal. The number of incorrectly classified clothing items are displayed outside that diagonal. Inspecting the output, it appears that the trained model incorrectly idenitifies quite a few clothing items as t-shirts and shirts.

For illustrative purposes we'll inspect a few of the incorrectly classified clothing items. While this probably won't provide you with any useful insights, it might satisfy your curiosity.

Next, we'll calculate precision, recall, and accuracy to formally quantify the model performance.

Calculate the precision scores¶

We use scikit-learn's sklearn.metrics.precision_score() method to calculate the precision for each of the 10 classes, using the correct labels and predicted labels as input. The precision quantifies the ability of the classifier not to label as positive a sample that is negative. For example, if many clothing items are incorrectly classified as shirts the precision score for shirts should be lower. A value of 1 is best and 0 the worst.

The precision score for shirts indicates that the model incorrectly identifies quite a few clothing items that are not shirts as shirts.

Calculate the recall¶

We use scikit-learn's sklearn.metrics.recall_score() method to calculate the precision for each of the 10 classes, using the correct labels and predicted labels as input. The recall quantifies the ability of the classifier find all positive examples. For example, a high recall value for trousers indicates that the model does well identifying trousers. A value of 1 is best and 0 the worst.

It appears that the trained model isn't very good yet at correctly identifying pullovers, coats, and shirts.

Calculate the overall accuracy¶

We use scikit-learn's sklearn.metrics.acuracy_score() method to calculate the accuracy score, using the correct labels and predicted labels as input.

Conclusions¶

Our trained model evaluation has identified a few weaknesses that have an impact on the overall performance of this model. Given the insights you could try the following to improve the model's prediction performance for poorly classified classes, such as shirts and t-shirts:

• Tune the model hyperparameters in order to improve performance on those classes, without degrading the performance for other classes.
• Collect more images for those classes to give the model more data to learn from.
• Apply data augmentation approaches. For image data, this can be creating more data artificially by doing “flips”, “rotations”, playing with color, brightness, etc in order to create more and “more difficult to classify” training examples in order to allow the model to learn more with the same amount of actual data.

Next steps¶

• Close this notebook.

Authors¶

This notebook was created by the Center for Open-Source Data & AI Technologies.

Copyright © 2019 IBM. This notebook and its source code are released under the terms of the MIT License.